Parsing Bond Responses From The Halo Infinite API

If you’ve been following my blog, you know that I’ve been fiddling quite a bit with the Halo Infinite API. To the point where I started writing my own wrapper for it.

As I was investigating the API, I’ve noticed something peculiar about the requests that were coming out of my machine and the responses I was getting.

Notice the Accept: application/x-bond-compact-binary header along with content that kind of looks like gibberish with the occasional nuggets of information embedded in them. I’ve never seen this before, but I worked a little bit with protocol buffers and this seemed oddly familiar. The x-bond-compact-binary was a signal that the content is likely a binary format that is used to reduce the amount of data that is sent over the wire. Doing a bit more research, I found out that this is nothing else but Microsoft’s take on protocol buffers - the Bond framework.

Having an idea of what it is, I got curious - just how much is 343 saving in terms of content size if they do try to squeeze as much performance as possible without extra overhead? If it’s anything like protocol buffers, my assumption was that the Bond format strips out most of the “metadata” around the content and just serializes it in a way that the data models on either end of the wire can understand, but beyond that - the data would be almost useless (since we wouldn’t know what we’re looking at).

To validate my size hypothesis, let’s take an API endpoint such as this:

https://gamecms-hacs.svc.halowaypoint.com:443
  /hi
  /multiplayer
  /file
  /Academy
  /BotCustomizationData.json?flight=<YOUR_FLIGHT_ID>

This endpoint is designed to get bot customization data and, just like many other API endpoints, accepts three types of formats in which data can be returned: application/json, application/xml (not all API endpoints actually support it - bit of a hit or miss here), and application/x-bond-compact-binary. It also returns quite a bit of information all at once, so it could be a good testing surface.

As a side-note - having all three formats available is incredibly convenient for reverse-engineering the API and understand what exactly it’s doing behind the scenes, but I digress. I first requested the JSON content for the response - a whopping 8,114 line response totaling to a rough 10.08KB with 21ms to response. Not terribly bad. There is no XML variant of this endpoint, so I didn’t have anything to compare here. The Bond-formatted content landed on my box in 17ms with a size of 9.23KB - an 8.432% savings in content volume. Not bad at all, especially since this adds up given just how many network requests Halo Infinite makes over the span of many multiplayer games - any savings in volume is good, and that also reduces the parsing overhead.

The challenge remained, though - how can I parse this out? I had no access to the underlying data model and it seemed that I needed it to be able to go from binary content to something a human can read. Apparently I was not the only one, because there is a GitHub issue requesting an implementation of a Fiddler inspector, which, by its nature, also requires the ability to deserialize Bond content without access to its data model.

Christopher Warrington provided the key insights as to how unstructured reading can go:

1. ReadStructBegin
2. ReadFieldBegin: this will tell you the field ordinal and its type

• if type is BT_STOP, be done with this struct
• if type is BT_STOP_BASE, continue reading fields but know that you’re one level lower in the hierarchy
• if type is a scalar, string, or blob, call the appropriate ReadFoo method
• if type is a container, call the appropriate ReadContainerBegin method: this will tell you the count of elements and the element type(s).
  • read count container elements, using the container type to know how to read each item
  • call ReadContainerEnd
• if type is a struct, recur

3. ReadFieldEnd
4. Loop back to step 2 until you see BT_STOP or BT_STOP_BASE.
5. ReadStructEnd

Additionally, Christopher called out two other important pieces of the process:

If you don’t read the field, you need to skip it.

ReadFieldBegin is “interpret the next bytes as if they were the preamble of the next field”, not “seek to next field and read it’s preamble”.

You need to make sure you initialize the reader with the right protocol and version. There’s no header or checksum in serialized data, so the reader can’t tell if you pointed a Fast Binary reader at Compact Binary stream. (Marshalled data has a header that indicates the protocol and version in the first four bytes.)

This all sounds like a reasonable way to start tackling the problem. Lucky for me, there is a Bond implementation in C# so I don’t need to do everything from scratch. My current implementation, following the algorithm above, is this (keep in mind that I do not read all field types - some are skipped):

using Bond;
using Bond.Protocols;
using System;
using System.IO;
using System.Text;

namespace ResponseReader
{
    class Program
    {
        static readonly StringBuilder globalComposite = new();

        static void Main(string[] args)
        {
            var ib = new Bond.IO.Unsafe.InputBuffer(File.ReadAllBytes("response.bin"));
            var cbr = new CompactBinaryReader<Bond.IO.Unsafe.InputBuffer>(ib, 2);

            ReadData(cbr);
            File.WriteAllText("log.txt", globalComposite.ToString());
        }

        static void Log(string value)
        {
            Console.WriteLine(value);
            globalComposite.AppendLine(value);
        }

        static void ReadData(CompactBinaryReader<Bond.IO.Unsafe.InputBuffer> reader)
        {
            reader.ReadStructBegin();

            BondDataType dataType;

            do
            {
                reader.ReadFieldBegin(out dataType, out ushort fieldId);
                Log($"Data type: {dataType}\tField ID: {fieldId}");

                DecideOnDataType(reader, dataType);

                reader.ReadFieldEnd();
            } while (dataType != BondDataType.BT_STOP);

            reader.ReadStructEnd();
        }

        private static void ReadContainer(CompactBinaryReader<Bond.IO.Unsafe.InputBuffer> reader)
        {
            reader.ReadContainerBegin(out int containerCounter, out BondDataType containerDataType);
            Log($"Reading container with item type: {containerDataType}\tItems in container: {containerCounter}");

            for (int i = 0; i < containerCounter; i++)
            {
                Log("Traversing list item: " + containerCounter);
                DecideOnDataType(reader, containerDataType);
            }

            Log("Done reading container.");

            reader.ReadContainerEnd();
        }

        private static void DecideOnDataType(CompactBinaryReader<Bond.IO.Unsafe.InputBuffer> reader, BondDataType dataType)
        {
            switch (dataType)
            {
                case BondDataType.BT_STRING:
                    {
                        var stringValue = reader.ReadString();
                        Log(stringValue);
                        break;
                    }
                case BondDataType.BT_LIST:
                    {
                        ReadContainer(reader);
                        break;
                    }
                case BondDataType.BT_STRUCT:
                    {
                        ReadData(reader);
                        break;
                    }
                case BondDataType.BT_BOOL:
                    {
                        var boolValue = reader.ReadBool();
                        Log(boolValue.ToString());
                        break;
                    }
                case BondDataType.BT_INT32:
                    {
                        var int32Value = reader.ReadInt32();
                        Log(int32Value.ToString());
                        break;
                    }
                case BondDataType.BT_DOUBLE:
                    {
                        double doubleValue = reader.ReadDouble();
                        Log(doubleValue.ToString());
                        break;
                    }
                case BondDataType.BT_FLOAT:
                    {
                        double floatValue = reader.ReadFloat();
                        Log(floatValue.ToString());
                        break;
                    }
                default:
                    if (dataType != BondDataType.BT_STOP)
                    {
                        reader.Skip(dataType);
                    }
                    break;
            }
        }
    }
}

One thing I’ll call out is that this was done using a bit of trial-and-error - pay attention to this line:

var cbr = 
  new CompactBinaryReader<Bond.IO.Unsafe.InputBuffer>(ib, 2);

Recall earlier that the content type that we worked with was application/x-bond-compact-binary. This was the first clue that told me that I likely should be using CompactBinaryReader (refer to compact binary encoding reference) as opposed to something else. The 2 parameter is the protocol version - again, this is just coming from my assumption that the compact binary format comes with the second version of the protocol based on the aforementioned compact binary encoding reference:

BOND_STATIC_CONSTEXPR uint16_t magic = COMPACT_PROTOCOL;
BOND_STATIC_CONSTEXPR uint16_t version = v2;

With this implementation I was able to take a response I got that was encoded in Bond format, and try and parse it out locally. How do you save a Bond response? Whatever tool you use for experimentation, such as Fiddler or Postman (or maybe you’re even writing your own) generally allows you to save the binary content by exporting it to a file.

Running the code on the sample response from the bot customization endpoint I called out earlier I seemed to get the right result that looks a bit less like gibberish:

To compare, I looked at the JSON response and selected a small chunk:

"RightShoulderPadPath": "Inventory/armor/shouldersright/009-001-reach-5501ff72.json",
"Emblems": [
    {
        "Path": "Inventory/armor/Emblems/013-001-73651e6b.json",
        "LocationId": 1242505775,
        "ConfigurationId": -322620398
    }
],

Then I tried to find the same chunk in the decoded response:

Traversing list item: 1
Inventory/armor/shouldersright/009-001-reach-5501ff72.json
Done reading container.
Data type: BT_LIST  Field ID: 13
Reading container with item type: BT_LIST Items in container: 1
Traversing list item: 1
Reading container with item type: BT_STRUCT Items in container: 1
Traversing list item: 1
Data type: BT_STRING  Field ID: 0
Inventory/armor/Emblems/013-001-73651e6b.json
Data type: BT_INT32 Field ID: 1
1242505775
Data type: BT_INT32 Field ID: 2
-322620398

The latter looks a bit more verbose purely because I’ve added quite a bit of logging to my code to make sure that I am able to diagnose any parsing issues. So far, though, it looks good, and I am able to read Bond responses. As you can tell, it would be much harder to interpret those without having the data model handy. There is no indication what any of the data means and you would just need to infer it from context.

A lot of it can likely be done from looking at content groups (e.g., you’re looking at lists of bot appearance configurations), but things like integer values that are flagged in the JSON response as LocationId and ConfigurationId would be impossible to know without prior knowledge.

All this being said, I hope this can give you a better idea of how to look at Bond content coming through the Halo API, and combined with JSON, and in select cases - XML responses, you can make more sense of what goes on between your computer and the Halo servers.